Information processing apparatus, information processing method, and program

ABSTRACT

An information processing apparatus ( 100 ) includes: a person detecting unit ( 102 ) that executes processing of detecting a person and a location with respect to each of a first image generated in a first camera ( 5   a ) and a second image generated in a second camera ( 5   b ); a coordinate generating unit ( 104 ) that generates coordinates of the location of the detected person; and a determination unit ( 106 ) that determines, by using a first location generated from the first image and a second location generated from the second image, whether the person detected from the first image and the person detected from the second image are the same person.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 17/435,487 filed on Sep. 1, 2021, which is aNational Stage Entry of PCT/JP2020/008661 filed on Mar. 2, 2020, whichclaims the benefit of priority from Japanese Patent Application2019-039015 filed on Mar. 4, 2019, the disclosures of all of which areincorporated in their entirety by reference herein.

TECHNICAL FIELD

The present invention relates to an information processing apparatus, aninformation processing method, and a program.

BACKGROUND ART

Many systems that determine a flow line where a plurality of personsmove in a predetermined space such as a store and a facility, analyzethe determined flow line, and execute a behavior analysis have beenproposed. A system described in Patent Document 1, for example,computes, by using an image captured by a camera, a flow situation of aperson or a prediction value of a flow situation. Further, when movementdirections of a plurality of persons are individually predicted, it isdifficult to highly accurately estimate, due to occlusion of a cameraand the like, a movement direction of a person who temporarilydisappears from an image. In this regard, the system described in PatentDocument 1 includes a means that extracts, based on movement informationindicating a movement history of each of a plurality of persons, atleast two persons inferred to move in relation to each other, a meansthat determines a relationship between the extracted at least twopersons, a means that determines, based on the determined relationshipinformation, whether the at least two persons are grouped, and a meansthat predicts a behavior of each of the at least two persons determinedbe grouped.

RELATED DOCUMENT Patent Document

-   [Patent Document 1] Japanese Patent Application Publication No.    2019-21019

SUMMARY OF THE INVENTION Technical Problem

As described in Patent Document 1, it is desirable to individually tracka flow line highly accurately in a predetermined space with respect to aplurality of persons.

The present invention has been made in view of the circumstancedescribed above, and an object of the present invention is to provide atechnique for individually performing a detection of locations of aplurality of persons in a predetermined space highly accurately andefficiently.

Solution to Problem

According to each of aspects of the present invention, in order to solvethe above-described problem, the following configuration is employed foreach aspect.

A first aspect relates to an information processing apparatus.

The information processing apparatus according to the first aspectincludes:

-   -   a person detecting unit that executes processing of detecting a        person and a location with respect to each of a first image        generated in a first camera and a second image generated in a        second camera;    -   a coordinate generating unit that generates coordinates of the        location of the detected person; and    -   a determination unit that determines, by using a first location        generated from the first image and a second location generated        from the second image, whether the person detected from the        first image and the person detected from the second image are        the same person.

A second aspect relates to an information processing method executed byat least one computer.

The information processing method executed by an information processingapparatus according to the second aspect includes:

-   -   executing processing of detecting a person and a location with        respect to each of a first image generated in a first camera and        a second image generated in a second camera;    -   generating coordinates of the location of the detected person;        and    -   determining, by using a first location generated from the first        image and a second location generated from the second image,        whether the person detected from the first image and the person        detected from the second image are the same person.

Note that, another aspect according to the present invention may be aprogram that causes at least one computer to execute the methodaccording to the second aspect described above and may be a storagemedium readable by a computer that stores such a program. The storagemedium includes a non-transitory, tangible medium.

The computer program includes a computer program code that causes, whenthe computer program is executed by a computer, the computer to executethe information processing method on an information processingapparatus.

Note that, any combination of the components described above and a formacquired by converting an expression according to the present inventionamong a method, an apparatus, a system, a storage medium, and a computerprogram are also effective as an aspect according to the presentinvention.

Further, various types of components according to the present inventionmay not necessarily exist individually and independently, and thefollowing aspects may be employable: a plurality of components areformed as one member; one component is formed with a plurality ofmembers; a certain component is a part of another component; a part of acertain component and a part of another component are overlapped; andthe like.

Further, while, in the method and the computer program according to thepresent invention, a plurality of steps are described in order, an orderof the description does not limit an order for executing a plurality ofsteps. Therefore, when the method and the computer program according tothe present invention are executed, the order of a plurality of stepscan be modified without posing an obstacle in content.

Further, it is not limited to execute a plurality of steps of the methodand the computer program according to the present invention at timingsdifferent individually. Therefore, a case where, while a certain step isexecuted, another step may occur, a case where an execution timing of acertain step and an execution timing of another step may be overlappedpartially or entirely, and another case may occur.

Advantageous Effects of Invention

According to the aspects described above, a technique for individuallyperforming a detection of locations of a plurality of persons in apredetermined space highly accurately and efficiently can be provided.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-described object, other objects, features, and advantages willbecome more apparent from preferred example embodiments described belowand the following accompanying drawings.

FIG. 1 is a diagram conceptually illustrating a configuration of aninformation processing system according to an example embodiment of thepresent invention.

FIG. 2 is a diagram illustrating a disposition example of a productshelving unit and a camera in which a store managed by the informationprocessing system according to the example embodiment is viewed fromabove.

FIG. 3 is a function block diagram illustrating a logical configurationof an information processing apparatus according to the exampleembodiment.

FIG. 4 is a diagram illustrating one example of a configuration of acomputer that achieves the information processing apparatus according tothe example embodiment.

FIGS. 5A to 5C are diagrams each illustrating one example of a datastructure of data stored in a storage apparatus connected to theinformation processing apparatus according to the example embodiment.

FIGS. 6A and 6B are diagrams each illustrating one example of a datastructure of data stored in the storage apparatus connected to theinformation processing apparatus according to the example embodiment.

FIG. 7 is a diagram for illustrating a first coordinate system of afirst camera and a second coordinate system of a second camera beingdifferent from the first camera.

FIGS. 8A and 8B are diagrams each for illustrating a flow line where aperson moves in a store.

FIGS. 9A and 9B are diagrams each illustrating one example of userdetection information stored in the storage apparatus.

FIG. 10 is a flowchart illustrating one example of an operation of theinformation processing apparatus according to the example embodiment.

FIGS. 11A and 11B are diagrams each illustrating one example of a datastructure of data stored in the storage apparatus connected to theinformation processing apparatus according to the example embodiment.

FIG. 12 is a diagram for illustrating person detection and integrationprocessing of the information processing apparatus according to theexample embodiment.

FIG. 13 is a diagram illustrating one example of user detectioninformation generated based on images of cameras different from eachother in the information processing apparatus according to the exampleembodiment.

FIG. 14 is a function block diagram illustrating a logical configurationof the information processing apparatus according to the exampleembodiment.

FIGS. 15A to 15F are diagrams each illustrating a pattern of a flow lineof each of a plurality of persons.

FIG. 16 is a flowchart illustrating one example of an operation of theinformation processing apparatus according to the example embodiment.

FIG. 17 is a diagram conceptually illustrating a configuration of theinformation processing system according to the example embodiment of theinvention.

FIG. 18 is a diagram illustrating one example of a data structure ofdata stored in the storage apparatus connected to the informationprocessing apparatus according to the example embodiment.

FIG. 19 is a function block diagram illustrating a logical configurationof the information processing apparatus according to the exampleembodiment.

FIG. 20 is a flowchart illustrating one example of an operation of theinformation processing apparatus according to the example embodiment.

FIG. 21 is a diagram illustrating one example of an image in a situationwhere a plurality of persons are present in front of a product shelvingunit of a store.

DESCRIPTION OF EMBODIMENTS

Hereinafter, example embodiments according to the present invention aredescribed by using the accompanying drawings. Note that, in all thedrawings, a similar component is assigned with a similar reference signand description therefor is not repeated, as appropriate. Note that, indrawings, a configuration of a portion which does not relate to anessence according to the present invention is omitted and therefore isnot illustrated.

First Example Embodiment

FIG. 1 is a diagram conceptually illustrating a configuration of aninformation processing system 1 according to an example embodiment ofthe present invention. FIG. 2 is a diagram illustrating a dispositionexample of a product shelving unit 22 and a camera 5 in a store

The information processing system 1 captures, by using a plurality ofcameras 5, an image of a plurality of persons in a predetermined spacesuch as a store and a facility, and tracks a flow line of each person byusing the captured image. The information processing system 1automatically makes settlement, for example, only based on a procedureof picking up a product from a product shelving unit and putting theproduct in a cart by a user having entered the store when leaving thestore 20. Further, a product may not be necessarily put in a cart, andsettlement may be automatically made even by the user's putting aproduct in their own bag and leaving without doing anything.

The information processing system 1 captures an image of a person havingvisited the store 20 by using the camera 5 installed in a vicinity of anentrance, determines the person by executing face authentication, andconfirms store-entering of the user. Thereafter, also, by using thecamera 5 installed for each product shelving unit in the store 20, animage of a person in front of a product shelving unit is captured, andthereby while a user is determined based on face authentication, a flowline indicating a movement path of the user in the store 20 is tracked.In this example, a user is identified based on face authentication andtracked, and therefore the user continues to be tracked as one user fromstore-entering to leaving.

However, a plurality of persons are present in an actual store 20, andtherefore there have been cases where in a middle of processing oftracking, while each person is identified, a flow line of the person andprocessing of determining a person based on image recognition by usingimages of a plurality of cameras 5, the person is erroneously recognizedas a different person and it is temporarily difficult to execute imagerecognition, resulting in difficulty in identifying each person andaccurately tracking a flow line.

The information processing system 1 according to the present exampleembodiment can track flow lines of a plurality of persons highlyaccurately and efficiently by using a plurality of cameras 5.

The information processing system 1 includes a server 40. The server 40is connected to a communication network 3 via a relay apparatus (notillustrated) such as a router. The server 40 is connected, via thecommunication network 3, to a plurality of cameras 5 a, 5 b, (referredto as a “camera 5” unless otherwise specifically discriminated).

Apparatuses on the communication network 3 can be connected via at leasteither of a wired manner and a wireless manner. A communication means isnot specifically limited and various forms are conceivable. For example,a communication means such as a wireless local area network (LAN), awired LAN. a universal serial bus (USB), a communication means viavarious types of public lines such as 4th generation (4G), long termevolution (LTE), worldwide interoperability for microwave access(WiMAX), and the like are employable. In the figure, one network isillustrated, but a plurality of different communication networks may becombined.

According to the present example embodiment, in order to identify aplurality of persons in a captured image, it is assumed that, forexample, image recognition processing using feature information of aface is executed, but another piece of biological information is usable.For example, feature information of at least one of a face, an iris, oran auricle is usable. Alternatively, information indicating at least oneof features including a gait of a person, a stature, a shoulder width, aratio of a body part, clothes (a shape, a color, a material, and thelike), a hair style (including a hair color), an adornment (a hat, eyeglasses, an accessary, and the like), belongings (a bag, an umbrella, astick, and the like), and the like is usable. Alternatively, featureinformation in which at least two of these pieces of feature informationare combined is usable.

The camera 5 includes a lens and an imaging element, such as a chargecoupled device (CCD) image sensor, and, for example, is a network camerasuch as an internet protocol (IP) camera. The network camera includes,for example, a wireless local area network (LAN) communication functionand is connected to the server 40 via the communication network 3, i.e.,a relay apparatus (not illustrated) such as a router. These cameras 5each may be a so-called surveillance camera in which a plurality ofsurveillance cameras are installed in a store and a facility. Then, thecamera 5 may include a mechanism of controlling a direction of a camerabody and a lens, executing zoom control, executing focusing, and thelike by following a motion of a person in harmony with the motion.

The example in FIG. 2 includes at least one camera 5 d that captures animage of a person entering the store 20 from an entrance (IN) of thestore 20, at least one camera 5 e that captures an image of a personleaving an exit (OUT), and a plurality of cameras 5 a, 5 b, and 5 c eachthat capture an image of a person in front of each of product shelvingunits (in the figure, product shelving units 22 a, 22 b, and 22 c areindicated but the product shelving unit 22 is referred to unlessotherwise specifically discriminated). Further, the camera 5 (notillustrated) that captures an image of a person in a room from ashieling of the store 20 may be included. Note that, FIG. 2 is adisposition diagram in which the store 20 is viewed from above, andtherefore, normally, when a person is in an upright position, a top of ahead can be viewed, but a person in the figure merely schematicallyillustrates a presence position of the person, and therefore it is notindicated that a person is in a supine position on a floor.

As illustrated in FIG. 2 , the cameras 5 a, 5 b, and 5 c haveimage-capture ranges 50 a, 50 b, and 50 c (hereinafter, referred to asan image-capture range 50 unless otherwise specifically discriminated),respectively. The image-capture range 50 a and the image-capture range50 b of the camera 5 a and the camera 5 b are partially overlapped, andtherefore either one of the cameras can capture an image of a personwhile the person moving from the product shelving unit 22 a to theproduct shelving unit 22 b. On the other hand, the image-capture range50 b and the image-capture range 50 c of the camera 5 b and the camera 5c are not overlapped, and therefore while a person moves from theproduct shelving unit 22 b to the product shelving unit 22 c, there is aregion where the person departs from the image-capture range 50, and asa result, it is understood that there is a period in which an image of aperson being tracked is not temporarily captured. Even when it isdifficult to continuously capture an image of a person sinceimage-capture ranges 50 are separate among a plurality of cameras 50 inthis manner, the information processing system 1 according to thepresent example embodiment can track a flow line of a person.Alternatively, even when a person being tracked slouches, turns his/herback, or hides behind another person or an object, the person can betracked.

The camera 5 and the server 40 may be directly connected or may beconnected via the communication network 3 or the like as describedabove. For example, image data captured by the camera 5 may be directlytransmitted to the server 40, and the server 40 may sequentially receivethe image data. Further, a storage unit (not illustrated) accessible byboth the camera 5 and the server 40 may be provided. In this case, imagedata captured by the camera 5 are stored in the storage unit. Then, theserver 40 may read image data from the storage unit. Alternatively, thecamera 5 may be configured in such a way as to include an imagerecognition processing function and transmit a recognition result to theserver 40.

Herein, image data may be at least either of a still image or a video. Adata format, a file format, a file size, an image size, a resolution ofan image, a frame rate of a video, and the like are not specificallylimited, and various forms are employable according to a specification,a standard, performance, and the like of the camera 5 and the server 40or a processing capacity, accuracy, and the like of an image analysis.And, at least one frame of the image data may be handled as an image tobe described later.

Further, with regard to a timing of transmission of an image from thecamera 5 to the server 40, real-time distribution, e.g., streamingdistribution is employable, or an image for a predetermined period maybe transmitted at a predetermined interval. The transmission timing maybe appropriately selected based on a memory capacity of the camera 5 andthe server 40, a communication capacity, an image processing ability, acommunication situation between the camera 5 and the server 40, and thelike, or may be modified in response to a situation change.

The server 40 is connected to a storage apparatus 45. The storageapparatus 45 stores image data received from the camera 5, featureinformation, tracking information (location information, and the like),and the like. The storage apparatus 45 may be an apparatus separate fromthe server 40, may be an apparatus included inside the server 40, or maybe a combination of the apparatuses.

FIG. 3 is a function block diagram illustrating a logical configurationof an information processing apparatus 100 according to the exampleembodiment of the present invention.

The information processing apparatus 100 includes a person detectingunit 102, a coordinate generating unit 104, and a determination unit106. The person detecting unit 102 executes processing of detecting aperson and a location with respect to each of a first image generated inthe first camera 5 a and a second image generated in the second camera 5b. The coordinate generating unit 104 generates coordinates of thelocation of the detected person. The determination unit 106 determines,by using a first location generated from the first image and a secondlocation generated from the second image, whether the person detectedfrom the first image and the person detected from the second image arethe same person.

According to the present example embodiment, the server 40 and thecamera 5 include at least one computer. Then, the information processingapparatus 100 is achieved based on a computer included in at leasteither one of the server 40 and the camera 5 in FIG. 1 .

FIG. 4 is a diagram illustrating one example of a configuration of acomputer 60 configuring the server 40 and the camera 5 that achieve theinformation processing apparatus 100 of the information processingsystem 1 according to the present example embodiment. The informationprocessing apparatus 100 may be configured by using a plurality ofcomputers 60 or may be achieved based on a virtual server.

The computer 60 includes a central processing unit (CPU) 62, a memory64, a program 80, being loaded on the memory 64, that achieves acomponent of each of servers and a user terminal, a storage 66 thatstores the program 80, an input/output (I/O) 68, and an interface(communication I/F 70) for communication network connection.

The CPU 62, the memory 64, the storage 66, the I/O 68, and thecommunication I/F 70 are connected to each other via a bus 69, and eachof the servers and the user terminal are controlled by the CPU 62.However, a method of mutually connecting the CPU 62 and the like is notlimited to bus connection.

The memory 64 is a memory such as a random access memory (RAM) and aread only memory (ROM). The storage 66 is a storage apparatus such as ahard disk, a solid state drive (SSD), or a memory card.

The storage 66 may be a memory such as a RAM and a ROM. The storage 66may be provided inside the computer 60, or may be provided, whenaccessible by the computer 60, outside the computer 60 and connected tothe computer 60 in a wired manner or a wireless manner. Alternatively,the storage 66 may be attachably/detachably provided in the computer 60.

The CPU 62 reads the program 80 stored in the storage 66 onto the memory64 and executes the read program 80, and thereby functions of units ofthe information processing apparatus 100 in FIG. 3 can be achieved.

The I/O 68 executes input/output control for data and a control signalbetween the computer 60 and another input/output apparatus. The anotherinput/output apparatus includes, for example, an input apparatus 72 suchas a keyboard, a touch panel, a mouse, and a microphone connected to theCPU 60, an output apparatus (not illustrated) such as a display(illustrated as a display apparatus 74 in the figure), a printer, and aspeaker, and an interface between these input/output apparatuses and thecomputer 60. Further, the I/O 68 may execute input/output control ofdata for a read or write apparatus (not illustrated) of another storagemedium.

The communication I/F 70 is a network connection interface forperforming communication between the computer 60 and an externalapparatus. The communication I/F 70 may be a network interface forconnection to a wired line or may be a network interface for connectionto a wireless line. For example, the computers 60 that achieve each ofthe servers and the user terminal are mutually connected based on thecommunication I/F 70 via the communication network 3.

Each of components of the information processing apparatus 100 accordingto the present example embodiment in FIG. 3 is achieved by anycombination of hardware and software of the computer 60 in FIG. 4 . And,it should be understood by those of ordinary skill in the art that anachieving method and an apparatus for the components include variousmodified examples. A function block diagram indicating the informationprocessing apparatus 100 according to each example embodiment does notindicate a configuration based on a hardware unit but indicates a blockbased on a logical function unit.

The CPU 62 of the computer 60 in FIG. 4 reads the program 80 stored inthe storage 66 onto the memory 64 and executes the read program, andthereby can achieve functions of units of the information processingapparatus 100 in FIG. 3 .

The computer program 80 according to the present example embodiment isdescribed in such a way as to cause the computer 60 for achieving theinformation processing apparatus 100 to execute a procedure of executingprocessing of detecting a person and a location with respect to each ofa first image generated in a first camera and a second image generatedin a second camera, a procedure of generating coordinates of thelocation of the detected person, and a procedure of determining, byusing a first location generated from the first image and a secondlocation generated from the second image, whether the person detectedfrom the first image and the person detected from the second image arethe same person.

The computer program 80 according to the present example embodiment maybe stored on a storage medium readable by the computer 60. The storagemedium is not specifically limited and various forms are conceivable.Further, the program 80 may be loaded onto the memory 64 of the computer60 from a storage medium, or may be downloaded, through the network 3,onto the computer 60 and loaded onto the memory 64.

The storage medium that stores the computer program 80 includes anon-transitory, tangible medium usable by the computer 60, and a programcode readable by the computer 60 is embedded in the medium. When thecomputer program 80 is executed on the computer 60, the computer 60 iscaused to execute an information processing method of achieving theinformation processing apparatus 100.

Referring back to FIG. 3 , each of components of the informationprocessing apparatus 100 is described in details.

The person detecting unit 102 detects, from a captured image, a faceregion of each person and/or feature information of a face, and detectspresence of a person. The person detecting unit 102 further extracts,when detecting a face region, feature information from the face region.Then, the person detecting unit 102 collates the extracted featureinformation with feature information (indicated as a face feature valuein FIG. 5A) previously registered in user information 120 in the figure.When a user is identified based on the collation, a identified user IDis associated with date and time information, and the resultinginformation is stored, as user identification information, in userdetection information 130 in FIG. 5B. As date and time information, atleast any one of an image-capture date and time of an image, a receptiondate and time of an image from the camera 5, a storage date and time inthe storage apparatus 45, and a storage date and time of a collationresult is exemplified. Further, date and time information is not alwaysrequired, and information capable of determining an image, e.g.,information capable of identifying the camera 5, image data, and animage frame is applicable.

On the other hand, when it is difficult to identify, based on thecollation, a user, identification information indicating a person unableto be identified, e.g., a tentative user ID is associated with date andtime information, and the resulting information is stored, as useridentification information, in user detection information 130 in FIG.5B. Further, feature information (indicated as a face feature value inFIG. 5C) of a face region of a person unmatched with feature informationof a face in the user information 120 is associated with the tentativeuser ID and date and time information, and the resulting information maybe stored in unregistered person information 132 in the figure.

Processing of identifying each person from an image may be repeatedlyexecuted at a predetermined cycle, and may be executed, for example, foreach image frame or may be executed at every predetermined time interval(e.g., every several seconds, or the like). The repeating cycle may beappropriately determined according to various conditions such as animage processing ability of the computer 60, the number of cameras 5,and a communication capacity of the communication network 3.

The person detecting unit 102 executes detection and image recognitionprocessing for a person with respect to image data of a plurality ofcameras 5 at a predetermined cycle, and the results are stored in thestorage apparatus 45 as user detection information 130 or unregisteredperson information 132.

A location in an image of each person detected in the person detectingunit 102 is indicated by at least a two-dimensional coordinate system.According to the present example embodiment, it is assumed that theabove-described location is further indicated by a three-dimensionalcoordinate system also including depth. The person detecting unit 102stores location information (coordinates) further in association withuser detection information 130. The user detection information 130 isillustrated in FIG. 6A. Further, the person detecting unit 102associates, with respect to each camera 5, user detection information130 with identification information (also referred to as a camera ID) ofa camera and stores the resulting information.

The coordinate generating unit 104 acquires, from the person detectingunit 102, information of a location in an image of each person detectedin the person detecting unit 102 and generates coordinates of thelocation of the person. It is assumed that the generated coordinates areat least two-dimensional and are, according to the present exampleembodiment, three-dimensional coordinates further including depthinformation. A generation method for coordinates is described later.

A coordinate system indicating a location of a person in an image ofeach of cameras 5 is different with respect to each camera 5.

FIG. 7 is a diagram for illustrating a first coordinate system V1 of thefirst camera 5 a and a second coordinate system V2 of the second camera5 b being different from the first camera 5 a. A first locationindicating a location of a person detected from a first image of thefirst camera 5 a is indicated by a first coordinate system V1 of thefirst camera 5 a. A second location indicating a location of a persondetected from a second image of the second camera 5 b is indicated by asecond coordinate system V2 of the second camera 5 b. As illustrated inthe figure, two coordinate systems are different from each other, andlocations and directions are also different. Note that, it is assumedthat a first coordinate system and a second coordinate system each arealso referred to as a local coordinate system.

The coordinate generating unit 104 converts a coordinate system for eachcamera 5 to a common coordinate system in the store 20. In other words,the coordinate generating unit 104 converts each of the first locationand the second location to coordinates in a third coordinate system (inthe figure, indicated as “global”). The third coordinate system is, butnot limited to, coordinates defined, for example, in a real space (e.g.,at least a part of a store).

A conversion rule between a local coordinate system and a globalcoordinate system is previously generated, for example, as follows.First, an object is disposed in each of a plurality of locationspreviously determined in a global coordinate system and images ofobjects are captured from each of cameras 5. Next, the images capturedby each of the cameras 5 are processed and locations on localcoordinates of the objects are determined. Then, from the determinedlocations and the locations on the global coordinate systems of theobjects, a conversion rule between the local coordinate system and theglobal coordinate system is set for each camera 5. The coordinategenerating unit 104 generates, for example, a first conversion rule froma first coordinate system to a third coordinate system and a secondconversion rule from a second coordinate system to the third coordinatesystem.

The information processing apparatus 100 further associates pieces oflocation information (global coordinates) indicated in the generatedthird coordinate system with user detection information 130 (FIG. 6B)and stores each of pieces of the resulting information in the storageapparatus 45.

The determination unit 106 determines whether persons are the same, byusing a location indicated in the third coordinate system, beinggenerated by the coordinate generating unit 104, stored in the userdetection information 130. Hereinafter, an example of determinationprocessing based on the determination unit 106 is described in detail.

When, for example, a user ID of a person detected from a first image anda user ID of a person detected from a second image, the persons beingdetected by the person detecting unit 102, are the same, the persons canbe estimated as the same person. However, a case where it is difficultto identify a person by image recognition processing is supposable.Alternatively, a case where a person is identified as a different personis supposable. There are situations, for example, as follows.

-   -   (a1) An image is unclear or it is difficult to extract a face        region, and therefore it is difficult to acquire, in image        recognition processing, a degree of similarity or a degree of        likelihood equal to or more than a threshold.    -   (a2) A person departs from the image-capture range 50 of the        camera 5, and therefore the person is not captured in an image.    -   (a3) Image recognition accuracy decreases due to presence of a        plurality of persons.    -   (a4) Distortion of an image, a computation error, or the like        occurs.

The determination unit 106 executes determination processing, based onaccumulated user detection information 130. In the accumulated userdetection information 130, information of each person is checked intime-series and determination processing is executed, and therebyinformation of a data record of a person having been temporarilyidentified as a different person, a person having not been identified,or the like can be updated.

A situation of (a1) described above is described as an example.

It is assumed that, for example, as illustrated in FIG. 8A, a certainperson (a member having a user ID of U0010) moves in a store from a timet1 to a time t3 in an arrow direction. FIG. 9A illustrates a scene wherelocation information of persons identified in an image of the firstcamera 5 a is stored in user detection information 130. FIG. 9Billustrates a scene where location information of persons identified inan image of the second camera 5 b is stored in user detectioninformation 130. Note that, each image-capture range 50 in the figuresis a range where a person is viewed from a side of a camera 5 installedon each product shelving unit 22.

At a time t1, a person is present in front of the first camera 5 a and auser is identified by image recognition processing based on featureinformation of a face, and as a result, a user ID is stored in the userdetection information 130 in FIG. 9A. On the other hand, the person isnot present in the image-capture range 50 b of the second camera 5 b andtherefore is not detected at the time t1 in FIG. 9B. Note that, in theexample of FIGS. 9A and 9B, there is one person and therefore only onepair of a user ID and location information is stored, but when aplurality of persons are detected, a plurality of pairs of user ID andlocation information for the plurality of persons are stored.

Next, at a time t2, a person is present in a region where theimage-capture range 50 a of the first camera 5 a and the image-capturerange 50 b of the second camera 5 b are overlapped. In image recognitionprocessing based on feature information of a face of a person extractedfrom an image captured by the second camera 5 b, a user is identified.On the other hand, it is assumed that the person turns his/her back onthe first camera 5 a and, in image recognition processing based onfeature information of a face of a person extracted from an imagecaptured by the first camera 5 a, it is difficult to identify a user (inthe figure, indicated as “NG”).

At a time t3, a person departs from the image-capture range 50 a of thefirst camera 5 a, and therefore no one is detected from an imagecaptured by the first camera 5 a. On the other hand, the person ispresent in the image-capture range 50 b of the second camera 5 b, andtherefore a user is identified by image recognition processing andstored in the user detection information 130 in FIG. 9B.

Further, coordinates are generated by the coordinate generating unit 104from a location of a person detected by the person detecting unit 102.Then, a storage processing unit stores the location information in eachpiece of the user detection information 130. The determination unit 106determines, as the same person, for example, persons having the samelocation information at the same time or located in a reference range.In the example illustrated in FIG. 8A, the determination unit 106 refersto the user detection information 130, and determines, based on timeinformation and location information, that a person having been detectedat the time t2 by the first camera 5 a but having not identified as auser is the same person as a person having a user ID of a U0010 detectedat the time t2 by the second camera 5 b.

Note that, in this example, in an image of the first camera 5 a, imagerecognition processing for a person fails, but even when imagerecognition processing for a person based on the image of the firstcamera 5 a succeeds and image recognition processing for a person in animage of the second camera 5 b fails, the determination unit 106 candetermine whether to be the same person, similarly to a case where imagerecognition processing for a person fails in an image of the firstcamera 5 a.

Note that, various determination criteria for the same person areconceivable and the following examples are cited without limitationthereto. Further, determination may be made by combining twodetermination criteria among the criteria.

-   -   (b1) A degree of matching, at the same time, between location        information of the user detection information 130 (FIG. 9A) of        the first camera 5 a and location information of the user        detection information 130 (FIG. 9B) of the second camera 5 b is        computed, and determination is made as the same person when the        degree indicates matching at a value equal to or more than a        predetermined value.    -   (b2) When location information of the user detection information        130 of the first camera 5 a and location information of the user        detection information 130 of the second camera 5 b at the same        time are not matched at a value equal to or more than a        predetermined value, it is assumed that, when user IDs are the        same, determination is made as the same person or as a        possibility of the same person.

An operation of the information processing apparatus 100 according tothe present example embodiment configured in this manner is describedbelow.

FIG. 10 is a flowchart illustrating one example of an operation of theinformation processing apparatus 100 according to the present exampleembodiment.

First, the person detecting unit 102 executes processing of detecting aperson and a location with respect to each of a first image generated bythe first camera 5 a and a second image generated by the second camera 5b (step S101). Next, the coordinate generating unit 104 generatescoordinates of the location of the detected person (step S103). Thedetermination unit 106 determines, by using a first location generatedfrom the first image and a second location generated from the secondimage, whether the person detected from the first image and the persondetected from the second image are the same person (step S105).

Herein, each of steps may be executed as a sequential procedure or maybe individually executed sequentially. In other words, after step S101and step S103 are executed as a sequential procedure, processing of S105may be executed based on accumulated user detection information 130after user detection information 130 to some extent is accumulated.

As described above, according to the present example embodiment, byusing a plurality of cameras 5, locations of a plurality of persons in apredetermined space may be individually detected highly accurately andefficiently.

Second Example Embodiment

An information processing apparatus 100 according to the present exampleembodiment is similar to the information processing apparatus 100according to the first example embodiment except that, as adetermination criterion for the same person, (c1) and (c2) describedbelow are further used.

-   -   (c1) When times of user detection information 130 of FIG. 9A and        FIG. 9B deviate from each other or when a person is not detected        at the same time, a location of a first person is estimated from        a history of a location of at least the first person, and the        estimation result is used. For example, a certain reference time        is defined and location information at the reference time of a        first person is estimated. Then, at the reference time, when a        degree of matching between the estimated location information of        the first person and a detection value of location information        or estimated location information of a second person satisfies a        reference, it is determined that the first person and the second        person are the same person.    -   (c2) In the case of (c1) described above, even when estimated        locations of persons are not matched at a value equal to or more        than a predetermined value, determination is made, when user IDs        are the same, as the same persons or as a possibility of the        same person.

The information processing apparatus 100 estimates, when, for example, aperson is temporarily absent between images of different cameras 5, amovement location of the person, based on movement vectors of persons,and determines whether persons of the images of the different cameras 5are the same.

As one example, a determination unit 106 determines whether persons arethe same, by using a movement vector of a person generated by using aplurality of first images, a time from a time when the person is notdetected in a first image to a time when a new person is detected in asecond image, and a location of the new person.

The determination unit 106 generates, by using a plurality of firstimages 52 a, a movement vector (or a movement direction and a movementvelocity) of each of persons, associates a user ID (or a tentative userID) with time information, and stores the resulting information in astorage apparatus 45 as flow line information 140 in FIG. 11A. As theflow line information 140, a movement vector is accumulated by beingassociated with time information every time being detected for eachperson.

As illustrated in FIG. 12 and FIG. 13 , in image recognition processingin the first image 52 a, a user A having a user ID as U0012 isidentified at a time t1 and a time t2. At a time t3, the user A havingbeen identified so far is not recognized. On the other hand, in imagerecognition processing in a second image 52 b, a person is not detecteduntil the times t1 and t2, but at the time t3, a person identified as auser B having a tentative user ID as X0005 appears.

In this example, the determination unit 106 computes a movement vectorfrom location information of the user A detected in the first image 52a. Then, the determination unit 106 computes a time after the user A isnot detected in the first image 52 a and until the user B is detected inthe second image 52 b, and estimates a location of a movementdestination of a user from the movement vector of the user A and thecomputed time. A movement amount is estimated, for example, bymultiplying a movement vector by a computed time, and the estimatedmovement amount is added to newest location information (e.g., locationinformation at the time t2) of the user A, and thereby a location of theuser A is estimated. The estimated location information(movement-destination location information) is stored in the storageapparatus 45 as estimated information 150 in FIG. 11B. As the estimatedinformation 150, every time location information is estimated for eachperson, estimated location information is accumulated in associationwith time information.

The determination unit 106 determines whether an estimated location anda location of the user B detected in the second image 52 b are matched,and makes determination, when being matched, as the same person. Thedetermination unit 106 computes, for example, a degree of matchingbetween an estimated location of the user A and a detected location ofthe user B, and determines, when the degree indicates matching at avalue equal to or more than a predetermined value, that the users arethe same person.

Further, an example of (c2) described above is described.

It is assumed that, as illustrated in FIG. 8B, a person (a member havinga user ID as U0010) further moves from a state of FIG. 8A and moves in astore in an arrow direction from a time t3 to a time t5. At the time t3,a person is present in an image-capture range 50 b of a second camera 5b, and therefore a user is identified by image recognition processingand the resulting information is stored in user detection information130 in FIG. 9B. However, at a time t4, a person departs from theimage-capture range 50 b of the second camera 5 b, and therefore theperson is not detected from an image captured by the second camera 5 b.On the other hand, a person departs from an image-captured range 50 c ofa neighboring camera 5 c, and therefore the person is not eitherdetected from an image captured by the camera 5 c.

Further, at a time t5, a person moves up to the image-capture range 50 cof the camera and therefore when a person is identified by imagerecognition processing as a member having a user ID as U0010, thedetermination unit 106 determines that the person detected in the camera5 b and the person detected in the camera 5 c are the same person.

As described above, according to the present example embodiment, when amovement location of a person is estimated based on movement vectors ofpersons, it can be determined whether persons detected based on imagesof a plurality of different cameras 5 are the same, and thereforesimilarly to the example embodiments described above, using a pluralityof cameras 5, locations of a plurality of persons in a predeterminedspace can be individually detected highly accurately and efficiently. Indetermination of a person based on a captured image, it can be avoidedthat, for example, it is difficult to determine a person since imagerecognition is not temporarily executable and detection accuracy of aperson location decreases.

Third Example Embodiment

FIG. 14 is a function block diagram illustrating a logical configurationof an information processing apparatus 100 according to the presentexample embodiment.

The information processing apparatus 100 according to the presentexample embodiment is similar to any of the example embodimentsdescribed above except that a configuration for estimating a flow lineof a person by using a movement vector of the person generated by usinga plurality of images is included.

The information processing apparatus 100 according to the presentexample embodiment includes, similarly to the information processingapparatus 100 in FIG. 3 , a person detecting unit 102, a coordinategenerating unit 104, and a determination unit 106 and further includes aflow-line estimation unit 112. Further, in the present figure, theinformation processing apparatus 100 includes a storage processing unit110.

The flow-line estimation unit 112 estimates, by using a movement vector(flow line information 140 in FIG. 11A) of a person generated by using aplurality of images captured in at least one of a plurality of cameras5, a flow line of the person. Then, the determination unit 106 maycompare estimated flow lines of persons and make determination as beingthe same person when a degree of matching with a flow line of adifferent person (different in user ID) has a value equal to or morethan a predetermined threshold.

Estimation processing for a movement destination based on thedetermination unit 106 according to the example embodiments describedabove may be executed by the flow-line estimation unit 112. Flow lineinformation estimated by the flow-line estimation unit 112 is stored inestimated information 150. In other words, flow line information isaccumulated as location information for each time with respect to eachuser.

The storage processing unit 110 integrates, when while in user detectioninformation 130, persons are stored as a plurality of persons differentfrom each other, the determination unit 106 determines that the personsare the same person, pieces of location information of the plurality ofpersons into location information of one person. In other words, thestorage processing unit 110 integrates pieces of location information ofpersons determined as the same person by the determination unit 106 intolocation information of one person and updates a data record of the userdetection information 130.

Herein, matching between flow lines is described by citing an example.It is assumed that figures in FIGS. 15A to 15F each indicate flow linesof different users in the same period.

In an example of FIG. 15A, a degree of matching between two flow linesbeing a flow line L1 and a flow line L2 is equal to or more than athreshold, and therefore the two users are determined by thedetermination unit 106 as being the same person and pieces of locationinformation of the two users are integrated. In an example of FIG. 15B,a degree of matching between two flow lines being a flow line L1 and aflow line L2 is less than a threshold, and therefore the two users aredetermined as being not the same person. Therefore, pieces of locationinformation are not integrated. In an example of FIG. 15C, a period ofone flow line L2 is a part of a period of the other flow line L1 and adegree of matching between the flow lines is equal to or more than athreshold with respect to the same period, and therefore the two usersare determined as being the same person and pieces of locationinformation of the two users are integrated.

In an example of FIG. 15D, a period of one flow line L2 is earlier instart time and termination time than a period of the other flow line L1,and therefore the periods of the two flow lines deviate. However, withregard to a matched period, a degree of matching between the two flowlines is equal to or more than a threshold, and therefore the two usersare determined as being the same person and pieces of locationinformation of the two users are integrated. In an example of FIG. 15E,while a degree of matching between period-anterior half portions of twoflow lines being a flow line L1 and a flow line L2 is equal to or morethan a threshold, a degree of matching between period-posterior halfportions is less than a threshold. In this case, it is conceivable thatan accompanying person acts separately from midway, and therefore twopersons are determined as being not the same person. Therefore, piecesof location information are not integrated. In an example of FIG. 15F,in a part of a period of each of two flow lines being a flow line L1 anda flow line L2, one flow line L1 and the other flow line L2 deviate inflow line to a large extent. A period deviating by a value equal to ormore than a predetermined value is equal to or more than a predeterminedvalue, and therefore a degree of matching between the two flow linesbeing the flow line L1 and the flow line L2 is less than a threshold andthe two users are determined as being not the same person. Therefore,pieces of location information are not integrated.

FIG. 16 is a flowchart illustrating one example of an operation of theinformation processing apparatus 100 according to the present exampleembodiment. A procedure of an operation illustrated by the present flowindicates details of processing for determination processing (step S105)described according to the example embodiments described above.

First, the storage processing unit 110 stores location information of adetected person in a storage apparatus 45 by person (step S111). Then,the flow-line estimation unit 112 estimates, by using a movement vectorof a person generated by using a plurality of images, a flow line of theperson (step S113). The estimated flow line is stored in the storageapparatus 45 as estimated information 150. Then, the determination unit106 determines whether a person detected as a different person is thesame by using the estimated flow line of the person (step S115).

The storage processing unit 110 integrates, when while being stored as aplurality of persons different from each other, the persons aredetermined as being the same person in step S115 (YES in step S117),pieces of location information of the persons into location informationof one person (step S119).

As described above, according to the present example embodiment, anadvantageous effect similar to the example embodiments described aboveis exerted and in addition, flow lines of a plurality of persons can betracked highly accurately and efficiently by using a plurality ofcameras 5.

When, for example, persons are identified, based on a plurality ofcameras 5, as different persons and flow lines are the same, based onflow lines of a plurality of persons being individually subjected toflow line tracking, it is determined whether to be the same person, andwhen determination is made as being the same person, integration intoinformation of one person can be executed. Even when, for example,persons are erroneously recognized, based on a plurality of cameras 5,as different persons, data can be integrated later as the same person,and therefore determination accuracy for a person and a flow line of aperson increases.

Fourth Example Embodiment

An information processing apparatus 100 according to the present exampleembodiment is similar to any of the example embodiments described aboveexcept that a configuration for executing settlement processing, as apurchased product of a customer, for a product picked up by a personbeing the customer in a store 20 is included. An information processingsystem 1 according to the present example embodiment provides a servicein which, in the store 20 or a facility such as a shopping mall(hereinafter, the store 20 is described as a representative), a customerbeing previously registered for use can do shopping based on so-calledwalk-through. Further, even for a user being not previously registeredfor use, product registration in a product list is made only by pickingup a product from a product shelving unit 22, and therefore paymentprocessing may be executed, based on the registered product list, at aleaving time.

FIG. 17 is a diagram conceptually illustrating a configuration of theinformation processing system 1 according to the present exampleembodiment.

According to the present example embodiment, the information processingsystem 1 further includes, in addition to the configuration of theinformation processing system 1 according to the example embodimentsdescribed above in FIG. 1 , a plurality of sensors 7 that each detectthat a product is picked up from the product shelving unit 22. Each ofthe sensor 7 is connected to a server 40 via a communication network 3.Further, the server 40 may be connected to a digital signage 9 via thecommunication network 3. In the digital signage 9, a guide for the store20, event information, a product, a company advertisement, and the likemay be displayed.

According to the present example embodiment, a person is a user usingthe store 20 and uses the store 20 by carrying a user terminal 30. Theuser terminal 30 is, for example, a smartphone, a tablet terminal, amobile phone, or a personal digital assistant (PDA), and is any mobileterminal including a communication function without specific limitation.Further, the user terminal 30 may be a possession of a user or may be,for example, a terminal rented to a user in the store 20. The userterminal 30 may be connectable to the server 40 or the Internet (notillustrated) via a gateway (GW) 11 and the communication network 3.

The user terminal 30 includes at least one computer 60 illustrated inFIG. 4 . A user is previously registered as a user in order to use aservice provided by the information processing system 1 according to thepresent example embodiment in the store 20. As a user registrationmethod, for example, by using a user terminal 30 of a user, a dedicatedapplication is started and the server 40 is connected or by using abrowser or the like, a predetermined website is accessed and a userregistration screen is displayed on the user terminal 30, and thereby aregistration can be made. A user previously sets authenticationinformation such as a user ID and a password, inputs, at a time of usinga service, the authentication information to a log-in screen by usingthe user terminal 30, and logs in the present information processingsystem 1. Note that also, according to the present example embodiment,the present system can be used without registration for use, similarlyto the example embodiments described above. In this case, similarly tothe example embodiments described above, by using feature information ofa face of a person, a person is identified.

Further, a user registers feature information for identifying a user byimage recognition processing. For example, by using a camera of the userterminal 30, an image of a face is captured and thereby image data maybe registered. The server 40 stores, in a storage apparatus as userinformation 120 (FIG. 5A), feature information of a face extracted froma registered face image in association with user identificationinformation.

Information to be registered is not limited to these, and personalinformation (a name, an address, a contact, age, gender, and the like)of a user is further included. Authentication information and updateinformation may be stored in association with user information 120.

Further, a user also registers card information of a credit card used incard settlement for a price of shopping. Settlement card informationreceived from a user is associated with user identification informationand is stored in the storage apparatus 45 as card information 220 inFIG. 18 . The card information includes information required for cardsettlement and includes information, for example, such as a card type, acard number, a password, a name, and an expiration period.

FIG. 19 is a function block diagram illustrating a logical configurationof the information processing apparatus 100 according to the presentexample embodiment.

The information processing apparatus 100 according to the presentexample embodiment further includes, in addition to the configuration ofany one of the information processing apparatuses 100 according to theexample embodiments described above, a product recognition unit 202, anpayment processing unit 204, and an output unit 206. In the figure, anexample of a combination with the information processing apparatus 100in FIG. 14 is illustrated.

The product recognition unit 202 infers or recognizes a product pickedup by a person from the product shelving unit 22, generates a productlist by associating the inferred or recognized product with a flow lineof the person, and stores the generated product list in the storageapparatus 45. The product recognition unit 202 infers or recognizes aproduct by using the sensor 7. The sensor 7 includes a depth sensor thatdetects a motion of a hand of a person located in front of the productshelving unit 22 and a weight sensor that detects a weight of an articleplaced on a shelf of the product shelving unit 22.

A method of inferring or recognizing a product by using the productrecognition unit 202 is as follows. First, the product recognition unit202 detects, by using a depth sensor 7, a motion of a hand of a personlocated in front of the product shelving unit 22, and determines, fromthe motion of the hand of the person, a location where an article ispicked up from a shelf. Then, the product recognition unit 202 infers,based on a weight change of the shelf of the product shelving unit 22detected by the weight sensor, a type of the article picked up from theshelf. Then, the product recognition unit 202 outputs, based onplanogram information, article determination information determining anarticle disposed on a shelf, e.g., an ID (code information in somecases) allocated to the article or an article name (e.g., a productname) of the article.

Alternatively, the product recognition unit 202 may collate, based onimage recognition processing, an article picked up by a person whoseimage is captured by a camera 5 with a feature value of an article imagepreviously registered, and determine the article, based on the result.Recognition processing of a product based on the product recognitionunit 202 may be repeated until settlement processing to be automaticallyexecuted when a person leaves from the store 20 is executed. Therefore,when a person picks up a plurality of products from a shelf, the productrecognition unit 202 additionally stores, in a product list by person, apicked-up product every time picked up. The product list is associatedwith a flow line of the person. However, the product list may beassociated with identification information of the person. In this case,identification information of a person is associated with flow lineinformation.

The payment processing unit 204 executes, based on a product list,payment processing for the person. The payment processing unit 204executes card settlement by using card information 220 in FIG. 18 of theperson. The settlement processing may be executed by using, as atrigger, a fact that, for example, an image of a person is captured by acamera 5 e installed at an exit (OUT) of the store 20 and facerecognition is made. The payment processing unit 204 may be an externalapparatus separate from the information processing apparatus 100. Aconfiguration in which payment processing is instructed from theinformation processing apparatus 100 to an external apparatus and anpayment processing result is received from the external apparatus may bemade.

Note that, before settlement processing, a customer can read a productlist by using a user terminal 30 of the customer. The user terminal 30of a customer transmits, for example, a user ID of the customer to theinformation processing apparatus 100. The output unit 206 transmits aproduct list of a person relevant to the user ID to the user terminal 30of the person. At that time, a product code of the product list may beconverted to a product name. Further, together with a product name, aprice of the product may be transmitted. In the latter case, a totalamount of money of registered products may be further transmitted to theuser terminal 30 of the customer.

The transmitted product list is displayed in the user terminal 30 of thecustomer. A screen of the terminal may include, for example, anoperation button for accepting a content and an amount of money of apurchased product and executing settlement processing, an operationbutton for making a content modification of a purchased product, and thelike.

The output unit 206 outputs an electronic receipt based on paymentprocessing. The electronic receipt may be transmitted to the userterminal 30 of each person. The electronic receipt is transmitted, forexample, to a previously registered address with respect to each person.Further, when a customer is registered as a user, the customer may login a website where customer information can be browsed and confirm apurchase history.

FIG. 20 is a flowchart illustrating one example of an operation of theinformation processing apparatus 100 according to the present exampleembodiment.

First, a person detecting unit 102 identifies a person, based on imagerecognition processing for a face, from an image of a person who entersthe store 20 captured by using a camera 5 d provided at an entrance (IN)of the store 20 (step S201). Then, the person detecting unit 102registers a user ID of a identified person or a feature value of a faceof a person unable to be identified, and a flow-line estimation unit 112starts tracking the person (step S203).

The product recognition unit 202 infers or recognizes a picked-upproduct from a motion of the person being tracked (step S205). Then, theproduct recognition unit 202 associates the inferred product with a flowline of the person, and generates a product list of a customer (stepS207). Step S205 and step S207 are repeatedly executed and are repeateduntil payment processing is executed.

The output unit 206 outputs a product list to the user terminal 30 (stepS209). When a customer accepts a purchase content or face authenticationfor a customer succeeds (YES in step S211), the payment processing unit204 executes, based on the product list, payment processing (herein,card settlement processing) for the person (step S213). The output unit206 transmits, when the settlement processing is completed (YES in stepS215), an electronic receipt to the user terminal 30 of the customer(step S217).

Until a content of the customer is accepted or face authenticationsucceeds in step S211, processing is repeated by returning to step S203.Further, when settlement processing is not completed in step S215,predetermined exceptional processing may be executed after a given timeelapses. The predetermined exceptional processing is processing ofreporting a fact that settlement processing is not completed, inaddition, a reason for incompletion, and the like to a customer or anadministrator of the store 20, and the like. The output unit 206transmits, to the user terminal 30, a message in that, for example,“Since an expiration date of a registered credit card has expired,settlement processing is not completed. Please, confirm credit cardinformation and update the information”, and the like.

Note that, according to the present example embodiment, flow-lineestimation processing and location-information integration processingdescribed according to the example embodiments described above may beexecuted based on a processing flow separately from the processing flowdescribed above. In other words, flow-line estimation processing ofreferring to location information for each person determined and storedbased on the processing flow and thereby interpolating the missing datadescribed above or correcting erroneously detected data,location-information integration processing of integrating pieces oflocation information of a plurality of persons, and the like areexecuted. The processing flow in FIG. 20 and a processing flow offlow-line estimation processing and location-information integrationprocessing may be executed in parallel on a real-time basis or may beexecuted at timings different from each other. Hereinafter, combinationsof timings for each of pieces of processing are exemplified.

-   -   (d1) Person determination (flow line estimation) and product        inference or product recognition are processed in parallel.

In this configuration, the information processing apparatus 100 furtherincludes a determination unit (not illustrated) that determines a personflow line to be associated with a recognized or inferred product. Thedetermination unit determines a flow line of a person associated withidentification information or feature information of a personidentified, by the person detecting unit 102, based on image recognitionof a person being present in front of a product shelving unit 22, andassociates the determined flow line of the person with identificationinformation of a product inferred or recognized as being picked up bythe person in front of the product shelving unit 22. The determinationunit determines a person, for example, by using, as a trigger, a factthat it is detected that a product is picked up by the person in frontof the product shelving unit 22 and the picked-up product is inferred orrecognized. The product recognition unit 202 may associate a flow lineof the determined person with product information.

-   -   (d2) Product inference or recognition is executed after person        determination is completed.

In this configuration, the product recognition unit 202 executesprocessing of detecting a motion of a hand of a person by using, as atrigger, a fact that, among persons being tracked in the store 20,determination of a person in front of the product shelving unit 22 basedon the person detecting unit 102 is completed, and executes productinference or recognition. Then, the product recognition unit 202associates a flow line of the determined person with identificationinformation of the inferred or recognized product.

-   -   (d3) Person determination is executed after product recognition        or inference is executed.

In this configuration, the product recognition unit 202 determines aproduct picked up by a person in front of the product shelving unit 22,associates the determined product with date and time information,product-picked-up location, and identification information of a product,and stores, as product-picked-up record information, the resultinginformation in the storage apparatus 45. Then, the informationprocessing apparatus 100 further includes a determination unit (notillustrated) that determines a person by collating date and timeinformation of the product-picked-up record information and theproduct-picked-up location with date and time information of a historyof location information of each person and location information. Theproduct recognition unit 202 associates a flow line of the persondetermined by the determination unit with identification information ofthe picked-up product, generates a product list, and stores thegenerated product list in the storage apparatus 45. In other words, inthis configuration, every time a product is picked up, only a product IDis fixed, addition to a product list is made pending, and the productlist is updated retroactively at a timing of completion of persondetermination.

As described above, according to the present example embodiment, at thestore 20, a person having entered the store 20 is identified based onface authentication by the person detecting unit 102, a flow line of theidentified person is tracked, a product picked up from the productshelving unit 22 is inferred or recognized by the product recognitionunit 202, the inferred or recognized product is associated with the flowline of the person, a product list is transmitted to the user terminal30 of the person before the person leaves from the store 20, paymentprocessing is executed by the payment processing unit 204, based on aninstruction for a settlement procedure after a customer confirms acontent or based on card information of a person determined bydetermining, based on face authentication, a person who leaves, and anelectronic receipt is transmitted to the user terminal 30, based on anpayment processing result, by the output unit 206. In this manner,according to the present example embodiment, product registration isautomatically executed only by picking up a product from the productshelving unit 22 of the store 20, and shopping can be finished only bymaking automatic settlement, based on a previously-registered creditcard, or an instruction issued for an payment procedure by specifying asettlement method by the user terminal 30, at a time of leaving thestore 20.

According to the above-described example embodiments, “acquisition”includes at least one of a matter that a local apparatus fetches data orinformation stored in another apparatus or a storage medium (activeacquisition) or a matter that data or information output from anotherapparatus is input to a local apparatus (passive acquisition). Anexample of active acquisition includes receiving, based on a request oran inquiry to another apparatus, a reply thereof, reading by accessinganother apparatus or a storage medium, and the like. Further, an exampleof passive acquisition includes receiving information being delivered(or transmitted, push-notified, or the like), and the like. Further,“acquisition” may be a matter that, among pieces of received data orinformation, a piece of data or information is selected and acquired, ora matter that, among pieces of delivered data or information, a piece ofdata or information is selected and received.

While example embodiments of the present invention have been describedwith reference to the drawings, these example embodiments are onlyexemplification of the present invention, and various configurationsother than the above-described configurations can also be employed.

For example, as illustrated in FIG. 21 , when, in front of the productshelving unit 22, a plurality of persons A, B, and C stand, a face ofthe person C being present behind the person A and the person B ishidden, and therefore there is a possibility that the face is notidentified based on image recognition processing. Even in such asituation, the person detecting unit 102 indicates, based on athree-dimensional coordinate system, location information of a personand thereby can detect a location of a depth of the person, andtherefore when a degree of matching has a value equal to or more than apredetermined threshold by comparing depth information of a location ofa detected person (a person unable to be identified as the person C)with depth information of an estimated flow line of the person C, adetermination unit 106 may determine a person whose face is unable to beauthenticated as the person C.

Further, the person detecting unit 102 may determine the number ofpersons being present in the store 20, based on the number of visitingpersons detected by the camera 5 d installed in an entrance (IN) of thestore 20 and the number of persons having left detected by the camera 5e installed in an exit (OUT) of the store 20. Then, the determinationunit 106 may determine the number of erroneously-determined persons,based on the determined number of persons and the number of personsdetected by cameras 5. In other words, when a value acquired bysubtracting the number of persons having left and the number of personsdetected by cameras 5 from the number of visiting persons is 0, apossibility of erroneous determination is low. On the other hand, in acase of a positive integer, a possibility that a missing person unableto be detected by the camera 5 is present is high. In a case of anegative integer, a possibility that one person is determined as adifferent person, based on erroneous determination, is high.

The determination unit 106 may make the determination described aboveand execute, when a possibility of erroneous determination is high,determination processing for whether to be the same person based on flowline information 140. Further, for example, the person detecting unit102 may detect each of the number of persons having entered the store 20and the number of persons having left and compute the number of personsto be currently present in the store 20, and the determination unit 106may compare the computed number of persons with the number of userswhose location information is stored in user detection information 130and execute, by using, as a trigger, a fact that a comparison resultindicates disagreement (i.e., a possibility of erroneous determinationin determination of a person is high), determination processing forwhether to be the same person based on flow line information 140.

At least two of the above-described example embodiments can be combinedwithin an extent that there is no conflict.

While the invention has been particularly shown and described withreference to example embodiments and examples thereof, the invention isnot limited to these example embodiments and examples. It will beunderstood by those of ordinary skill in the art that various changes inform and details may be made therein without departing from the scope ofthe present invention.

Note that, according to the present invention, when information relatingto a user is acquired or used, this should be legally executed.

The whole or part of the example embodiments described above can bedescribed as, but not limited to, the following supplementary notes.

-   -   1. An information processing apparatus including:        -   a person detecting unit that executes processing of            detecting a person and a location with respect to each of a            first image generated in a first camera and a second image            generated in a second camera;        -   a coordinate generating unit that generates coordinates of            the location of the detected person; and        -   a determination unit that determines, by using a first            location generated from the first image and a second            location generated from the second image, whether the person            detected from the first image and the person detected from            the second image are the same person.    -   2. The information processing apparatus according to        supplementary note 1, wherein the determination unit determines        whether the persons are the same, by using a movement vector of        the person generated by using a plurality of the first images, a        time from a time when the person is not detected in the first        image to a time when a new person is detected in the second        image, and a location of the new person.    -   3. The information processing apparatus according to        supplementary note 1 or 2, further including        -   a flow-line estimation unit that estimates a flow line of            the person by using a movement vector of the person            generated by using a plurality of images generated in at            least either of the first camera and the second camera,            wherein        -   the determination unit determines, by using the flow line of            the person estimated by the flow-line estimation unit,            whether a person detected as a different person is the same.    -   4. The information processing apparatus according to any one of        supplementary notes 1 to 3, further including        -   a storage processing unit that stores location information            of the person detected by the person detecting unit in a            storage unit by person, wherein        -   the storage processing unit integrates, when, while being            stored as a plurality of persons different from each other            in the storage unit, the plurality of persons are determined            as being the same person in the determination unit, pieces            of location information of the plurality of persons into            location information of one person.    -   5. The information processing apparatus according to any one of        supplementary notes 1 to 4, further including:        -   a flow-line estimation unit that estimates a flow line of            the person by using a movement vector of the person            generated by using a plurality of the first images;        -   a product recognition unit that infers or recognizes a            product picked up by the person from a product shelf,            associates the inferred or recognized product with the flow            line of the person, generates a product list, and stores the            generated product list in a storage unit;        -   an payment processing unit that executes, based on the            product list, payment processing for the person; and        -   an output unit that outputs an electronic receipt based on            the payment processing.    -   6. The information processing apparatus according to any one of        supplementary notes 1 to further including        -   a storage processing unit that stores location information            of the person detected by the person detection unit in a            storage unit by person, wherein        -   the person detection unit computes, from the number of            persons having entered a target region for person tracking            and the number of persons having left, the number of persons            to be present in the target region, and        -   the determination unit executes processing of comparing the            computed number of persons with the number of persons whose            location information is stored in the storage unit and            determining whether the person is the same by using, as a            trigger, a fact that a comparison result indicates            disagreement.    -   7. An information processing method executed by an information        processing apparatus, the method including:        -   executing processing of detecting a person and a location            with respect to each of a first image generated in a first            camera and a second image generated in a second camera;        -   generating coordinates of the location of the detected            person; and        -   determining, by using a first location generated from the            first image and a second location generated from the second            image, whether the person detected from the first image and            the person detected from the second image are the same            person.    -   8. The information processing method executed by the information        processing apparatus according to supplementary note 7, the        method further including, determining whether the persons are a        same, by using a movement vector of the person generated by        using a plurality of the first images, a time from a time when        the person is not detected in the first image to a time when a        new person is detected in the second image, and a location of        the new person.    -   9. The information processing method executed by the information        processing apparatus according to supplementary note 7 or 8, the        method further including:        -   estimating a flow line of the person by using a movement            vector of the person generated by using a plurality of            images generated in at least either of the first camera and            the second camera; and        -   determining, by using the estimated flow line of the person,            whether a person detected as a different person is the same.    -   10. The information processing method executed by the        information processing apparatus according to any one of        supplementary notes 7 to 9, the method further including:        storing location information of the detected person in a storage        apparatus by person; and integrating, when, while being stored        as a plurality of persons different from each other in the        storage apparatus, the plurality of persons are determined as        being the same person in the determination, pieces of location        information of the plurality of persons into location        information of one person.    -   11. The information processing method executed by the        information processing apparatus according to any one of        supplementary notes 7 to 10, the method further including:        -   estimating a flow line of the person by using a movement            vector of the person generated by using a plurality of the            first images;        -   inferring or recognizing a product picked up by the person            from a product shelf, associating the inferred or recognized            product with the flow line of the person, generating a            product list, and storing the generated product list in a            storage apparatus;        -   executing, based on the product list, payment processing for            the person; and        -   outputting an electronic receipt based on the payment            processing.    -   12. The information processing method executed by the        information processing apparatus according to any one of        supplementary notes 7 to 11, the method further including:        -   storing location information of the detected person in a            storage apparatus by person;        -   computing, from the number of persons having entered a            target region for person tracking and the number of persons            having left, the number of persons to be present in the            target region; and        -   executing processing of comparing the computed number of            persons with the number of persons whose location            information is stored in the storage apparatus and            determining whether the person is the same by using, as a            trigger, a fact that a comparison result indicates            disagreement.    -   13. A program for causing a computer to execute:        -   a procedure of executing processing of detecting a person            and a location with respect to each of a first image            generated in a first camera and a second image generated in            a second camera;        -   a procedure of generating coordinates of the location of the            detected person; and        -   a procedure of determining, by using a first location            generated from the first image and a second location            generated from the second image, whether the person detected            from the first image and the person detected from the second            image are the same person.    -   14. The program according to supplementary note 13, for causing        a computer to further execute        -   a procedure of determining whether the persons are the same,            by using a movement vector of the person generated by using            a plurality of the first images, a time from a time when the            person is not detected in the first image to a time when a            new person is detected in the second image, and a location            of the new person.    -   15. The program according to supplementary note 13 or 14, for        causing a computer to further execute:        -   a procedure of estimating a flow line of the person by using            a movement vector of the person generated by using a            plurality of images generated in at least either of the            first camera and the second camera; and        -   a procedure of determining, by using the estimated flow line            of the person, whether a person detected as a different            person is the same.    -   16. The program according to any one of supplementary notes 13        to 15, for causing a computer to further execute:        -   a procedure of storing location information of the detected            person in a storage apparatus by person; and        -   a procedure of integrating, when, while being stored as a            plurality of persons different from each other in the            storage apparatus, the plurality of persons are determined            as being the same person in the procedure of the            determination, pieces of location information of the            plurality of persons into location information of one            person.    -   17. The program according to any one of supplementary notes 13        to 16, for causing a computer to further execute:        -   a procedure of estimating a flow line of the person by using            a movement vector of the person generated by using a            plurality of the first images;        -   a procedure of inferring or recognizing a product picked up            by the person from a product shelf, associating the inferred            or recognized product with the flow line of the person,            generating a product list, and storing the generated product            list in a storage apparatus;        -   a procedure of executing, based on the product list, payment            processing for the person; and        -   a procedure of outputting an electronic receipt based on the            payment processing.    -   18. The program according to any one of supplementary notes 13        to 17, for causing a computer to further execute:        -   a procedure of storing location information of the detected            person in a storage apparatus by person;        -   a procedure of computing, from the number of persons having            entered a target region for person tracking and the number            of persons having left, the number of persons to be present            in the target region; and        -   a procedure of executing processing of comparing the            computed number of persons with the number of persons whose            location information is stored in the storage apparatus and            determining whether the person is the same by using, as a            trigger, a fact that a comparison result indicates            disagreement.

This application is based upon and claims the benefit of priority fromJapanese patent application No. 2019-039015, filed on Mar. 4, 2019, thedisclosure of which is incorporated herein in its entirety by reference.

1. An information processing apparatus comprising: at least one memoryconfigured to store instructions; and at least one processor configuredto execute the instructions to: estimate a flow line of a person using aplurality of images; recognize a product picked up by the person from aproduct shelf; generate a product list indicating the recognizedproduct; associate the product list with the estimated flow line of theperson; and execute, based on the product list, payment processing forthe person.
 2. The information processing apparatus according to claim1, wherein the at least one processor is further configured to executethe instructions to: estimate the flow line of the person using amovement vector of the person generated from the plurality of images. 3.The information processing apparatus according to claim 1, wherein theat least one processor is further configured to execute the instructionsto: initiate association of the product list and the flow line of theperson upon recognition of the product picked up by the person;
 4. Theinformation processing apparatus according to claim 1, wherein the atleast one processor is further configured to execute the instructionsto: initiate recognition of the product picked up of the person uponestimation of the flow line of the person.
 5. The information processingapparatus according to claim 1, wherein the at least one processor isfurther configured to execute the instructions to: initiate associationof the product list and the flow line information upon estimation of theflow line.
 6. The information processing apparatus according to claim 1,wherein the at least one processor is further configured to execute theinstructions to: initiate estimation of the flow line of the person upondetection of the person by performing face recognition of an imagecaptured by a camera installed at an entrance of a store.
 7. Theinformation processing apparatus according to claim 1, wherein the atleast one processor is further configured to execute the instructionsto: initiate execution of the payment processing of the person upondetection of the person by performing face recognition of an imagecaptured by a camera installed at an exit of a store.
 8. The informationprocessing apparatus according to claim 1, wherein the at least oneprocessor is further configured to execute the instructions to: transmitthe product list of the person to a terminal of the person before thepayment processing; and initiate execution of the payment processing ofthe person upon receiving an operation for accepting the product listfrom the terminal of the person.
 9. An information processing methodexecuted by at least one computer, wherein the method comprises:estimating a flow line of a person using a plurality of images;recognizing a product picked up by the person from a product shelf;generating a product list indicating the recognized product; associatingthe product list with the estimated flow line of the person; andexecuting, based on the product list, payment processing for the person.10. A non-transitory computer-readable storage medium storing a programcausing a computer to execute: a procedure of estimating a flow line ofa person using a plurality of images; a procedure of recognizing aproduct picked up by the person from a product shelf; a procedure ofgenerating a product list indicating the recognized product; a procedureof associating the product list with the estimated flow line of theperson; and a procedure of executing, based on the product list, paymentprocessing for the person.